Learning in Volatile Environments With the Bayes Factor Surprise

Abstract Surprise-based learning allows agents to rapidly adapt nonstationary stochastic environments characterized by sudden changes. We show that exact Bayesian inference in a hierarchical model gives rise surprise-modulated trade-off between forgetting old observations and integrating them with the new ones. The modulation depends on probability ratio, which we call Bayes Factor Surprise, tests prior belief against current belief. demonstrate several existing approximate algorithms, Surprise modulates rate of adaptation observations. derive three novel surprise-based one family particle filters, variational learning, message passing, have constant scaling observation sequence length particularly simple update dynamics for any distribution exponential family. Empirical results these algorithms estimate parameters better than alternative approaches reach levels performance comparable computationally more expensive algorithms. is related but different from Shannon Surprise. In two hypothetical experiments, make testable predictions physiological indicators dissociate theoretical insight casting various as well proposed online may be applied analysis animal human behavior reinforcement environments.